Softmax policy gradient methods can take exponential time to converge

Abstract The softmax policy gradient (PG) method, which performs ascent under parameterization, is arguably one of the de facto implementations optimization in modern reinforcement learning. For $$\gamma $$ γ -discounted infinite-horizon tabular Markov decision processes (MDPs), remarkable progress has recently been achieved towards establishing global convergence PG methods finding a near-optimal policy. However, prior results fall short delineating clear dependencies rates on salient parameters such as cardinality state space $${\mathcal {S}}$$ S and effective horizon $$\frac{1}{1-\gamma }$$ 1 - , both could be excessively large. In this paper, we deliver pessimistic message regarding iteration complexity methods, despite assuming access to exact computation. Specifically, demonstrate that method with stepsize $$\eta η can take $$\begin{aligned} \frac{1}{\eta } |{\mathcal {S}}|^{2^{\Omega \big (\frac{1}{1-\gamma }\big )}} ~\text {iterations} \end{aligned}$$ | 2 Ω ( ) iterations converge, even presence benign initialization an initial distribution amenable exploration (so mismatch coefficient not exceedingly large). This accomplished by characterizing algorithmic dynamics over carefully-constructed MDP containing only three actions. Our exponential lower bound hints at necessity carefully adjusting update rules or enforcing proper regularization accelerating methods.